Skip to content

Bug in to_datetime raising ValueError with None and NaT and more than 50 elements #41006

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 20, 2021

Conversation

phofl
Copy link
Member

@phofl phofl commented Apr 17, 2021

None and NaT are different for unique while convert_listlike casts None to NaT, hence causing dups
Not sure if we could do something better.

@phofl phofl added Dtype Conversions Unexpected or buggy dtype conversions Datetime Datetime data dtype labels Apr 17, 2021
def test_convert_object_to_datetime_with_cache(self):
# GH#39882
ser = pd.Series(
[None] + [pd.NaT] * 50 + [pd.Timestamp("2012-07-26")], dtype="object"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could make 50 parameter in tools.datetimes, ensures that the test stays valid even if that gets changed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Like this?

@@ -225,3 +227,16 @@ def test_convert_bool_dtype(self):
# GH32287
df = pd.DataFrame({"A": pd.array([True])})
tm.assert_frame_equal(df, df.convert_dtypes())

def test_convert_object_to_datetime_with_cache(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you put this with the other to_datetime tests: pandas/tests/tools/test_to_datetime.py; there a number of caching tests if you can locate nearby

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved, hope the location is ok, cache tests are everywhere there :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep lgtm. lots of tests :->

@phofl
Copy link
Member Author

phofl commented Apr 20, 2021

@jreback greenish

@jreback jreback added this to the 1.3 milestone Apr 20, 2021
@jreback jreback merged commit 54bd5cd into pandas-dev:master Apr 20, 2021
@jreback
Copy link
Contributor

jreback commented Apr 20, 2021

thanks @phofl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Datetime Datetime data dtype Dtype Conversions Unexpected or buggy dtype conversions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: InvalidIndexError: Reindexing only valid with uniquely valued Index objects on to_datetime
3 participants